Imaging system

ABSTRACT

According to an embodiment, an imaging system includes an image sensor, an imaging lens, a microlens array, an irradiator, a distance information acquiring unit, and a controller. The microlens array includes multiple microlenses arranged with a predetermined pitch, the microlenses being respectively associated with pixel blocks. The irradiator emits light to project a pattern onto an object. The distance information acquiring unit acquires information on the distance in the depth direction to the object on the basis of a signal resulting from photoelectric conversion performed by the image sensor. The controller controls the irradiator so that images contained in a pattern that is reflected by the object and scaled down on the image sensor by the imaging lens and the microlenses are smaller than the arrangement pitch of images each formed on the image sensor by each microlens and larger than twice the pixel.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is based upon and claims the benefit of priority fromJapanese Patent Application No. 2014-059087, filed on Mar. 20, 2014; theentire contents of which are incorporated herein by reference.

FIELD

Embodiments described herein relate generally to an imaging system.

BACKGROUND

Imaging technologies capable of acquiring a distance in the depthdirection to an object (a depth map) as two-dimensional informationinclude various techniques such as a technology of measuring theintensity and the return time of light reflected by an object by usingreference light, a stereo distance measuring technology using multiplecameras, and the like. Depth map information allows more advance objectrecognition than image information acquired from normal cameras, andthere is therefore a growing need for such depth map information asadditional input information in relatively inexpensive products such ashome electric appliances, game products, and industrial products.

Furthermore, among distance imaging techniques, there is known imagingdevices with a compound-eye structure including an imaging lens that isa structure capable of acquiring a number of parallaxes by using asingle camera and allowing distance measurement based on triangulation.

Sensor units such as cameras mounted on terminal devices such asportable terminals and cleaning robots are required to be capable ofacquiring high-resolution, two-dimensional visible images, small and lowin height (thin), and low in power consumption. Furthermore, in thefuture, new ways of using imaging modules with additional sophisticatedfunctions such as gesture input and depth map acquisition are requiredin addition to visible image acquisition.

Cameras having a microlens array and multi-parallax passive depth mapcameras such as a multiple-camera setup of related art can estimatedistance without light sources (lasers, LEDs, etc.) and are thereforesuitable for low power consuming, small devices driven by batteries, butare disadvantageous in principle in that distance cannot be measuredwhen an object has no texture (difference in luminance).

In the meantime, when a system including active illumination means isused in a multi-parallax system such as a stereo camera, calibration ofalignment between cameras and alignment between a camera and the activeillumination means is required, and there is a disadvantage thatoccurrence of misalignment results in degradation in accuracy.Furthermore, when multiple systems are used, there is a disadvantagethat a light source pattern emitted by one device may interfere with alight source pattern from another device, which causes an error.

BRIEF DESCRIPTION OF THE DRAWINGS

The patent or application file contains at least one drawing executed incolor. Copies of this patent or patent application publication withcolor drawing(s) will be provided by the Office upon request and paymentof the necessary fee. A more complete appreciation of the invention andmany of the attendant advantages thereof will be readily obtained as thesame becomes better understood by reference to the following detaileddescription when considered in connection with the accompanyingdrawings, wherein:

FIG. 1 is a configuration diagram illustrating a configuration of animaging system according to an embodiment;

FIG. 2 is a diagram schematically illustrating a cross section of animaging module;

FIGS. 3A and 3B are diagrams illustrating geometric optical relations inan optical system of an imaging device;

FIG. 4 is a diagram schematically illustrating a method ofreconstructing a two-dimensional visible image by a reconstructing unit;

FIGS. 5A and 5B are diagrams schematically illustrating templatematching of a ML image;

FIG. 6 is a diagram illustrating an example of relative positions of theimaging system and an object according to the embodiment;

FIGS. 7A and 7D illustrate ML images with and without a pattern,respectively, FIGS. 7B and 7E illustrate reconstructed images with andwithout a pattern, respectively, and FIGS. 7C and 7F illustrate depthmaps with and without a pattern, respectively;

FIGS. 8A and 8B illustrate part of an ML image (multiple ML images) andpart of a reconstructed image, respectively;

FIG. 9 illustrates an ML image with a high black occupancy ratio in apattern and an ML image with a low black occupancy ratio in a pattern;

FIG. 10A and 10B are schematic diagrams of pattern shapes suitable forthe imaging system according to the embodiment;

FIG. 11 illustrates a pattern combining random patterns with differentperiods;

FIG. 12 is a diagram illustrating a first example of a periodic patterncombining multiple periods;

FIG. 13 is a diagram illustrating a second example of a periodic patterncombining multiple periods;

FIG. 14 is a diagram illustrating a third example of a periodic patterncombining multiple periods;

FIG. 15 is a flowchart illustrating an example of operation of theimaging system according to the embodiment;

FIGS. 16A to 16C are diagrams illustrating a first modified example ofthe imaging system according to the embodiment; and

FIGS. 17A and 17B are diagrams illustrating a second modified example ofthe imaging system according to the embodiment.

DETAILED DESCRIPTION

According to an embodiment, an imaging system includes an image sensor,an imaging lens, a microlens array, an irradiator, a distanceinformation acquiring unit, and a controller. The image sensor includesmultiple pixel blocks each containing multiple pixels configured tocarry out photoelectric conversion. The imaging lens focuses light froman object onto a virtual imaging plane. The microlens array is providedbetween the image sensor and the imaging lens and includes multiplemicrolenses arranged with a predetermined pitch. The microlenses arerespectively associated with the pixel blocks. The irradiator emitslight to project a pattern onto the object. The distance informationacquiring unit acquires information on distance in a depth direction tothe object on the basis of a signal resulting from photoelectricconversion performed by the image sensor. The controller controls theirradiator so that the pattern formed on the image sensor satisfies thefollowing expressions (1) and (2):

fp=Fpt×M×N   (1), and

1/L _(ML) <Fpt×M×N<1/2dpix   (2),

where fp represents a frequency of an image formed on the image sensor,Fpt represents a frequency of the pattern, M represents a magnificationof the imaging lens, N represents a magnification of the microlenses,L_(ML) represents a distance between the microlenses, and dpixrepresents a pixel size.

An imaging system according to an embodiment will be described belowwith reference to the accompanying drawings.

Embodiment

FIG. 1 is a configuration diagram illustrating a configuration of animaging system 1 according to the embodiment. The imaging system 1includes an imaging device 10 and an irradiator 12, and is capable ofoutputting a visible image and a depth map. The imaging system 1 mayinclude multiple irradiators 12 (see FIG. 6).

The imaging device 10 includes an imaging module 2, an image signalprocessor (ISP) 4, a controller 50, and a determining unit 52. Theimaging module 2 includes an image sensor 20, an imaging optical system(imaging lens) 22, a microlens array (MLA) 24, and an imaging circuit26.

The image sensor 20 functions as an element that converts light capturedby the imaging optical system 22 into a signal charge in units ofpixels, and includes multiple pixels (photodiodes, for example, whichare photoelectric conversion elements) arranged in a two-dimensionalarray. The imaging optical system 22 functions as an imaging opticalsystem that captures light from an object into the image sensor 20. Theimage sensor includes multiple pixel blocks arranged in atwo-dimensional array. Each of the pixel blocks includes multiple pixelsarranged in a two-dimensional array.

The microlens array 24 is a micro-optical system such as a microlensarray or a prism having multiple microlenses, for example. The microlensarray 24 functions as an optical system that scales down and re-images agroup of light beams focused on an imaging surface by the imagingoptical system 22 to pixel blocks associated with individual microlenses(MLs).

The imaging circuit 26 includes a drive circuit part (not illustrated)configured to drive the pixels of the image sensor 20, and a pixelsignal processing circuit part (not illustrated) configured to processsignals output from pixel regions. The drive circuit part includes avertical selection circuit configured to sequentially select pixels tobe driven in units of horizontal lines (rows) in the vertical direction,an horizontal selection circuit configured to sequentially select pixelsin units of columns, and a timing generator (TG) circuit configured todrive the selection circuits by various pulses, for example. The pixelsignal processing circuit part includes an A/D conversion circuitconfigured to convert analog electric signals from the pixel regionsinto digital signals, a gain adjustment/amplification circuit configuredto carry out gain adjustment and amplification, and a digital signalprocessing circuit configured to carry out correction of digital signalsand the like, for example.

The ISP 4 includes a module interface (I/F) (camera module interface)40, an image capturing unit 41, a signal processor 42, a reconstructingunit 43, a depth map generator 44, and a driver interface (I/F) 45. Themodule I/F 40 is an interface that receives a RAW image taken by theimaging module 2 and outputs the RAW image to the image capturing unit41. The image capturing unit 41 captures the RAW image taken by theimaging module 2 via the module I/F 40.

The signal processor 42 carries out signal processing on the RAW imagecaptured by the image capturing unit 41 to generate images (microlensimages: ML images) each imaged for each microlens, for example. Thesignal processor 42 also outputs data (multiple ML images: acompound-eye image) on which signal processing is carried out to thereconstructing unit 43 and the controller 50.

The reconstructing unit 43 uses the ML images (compound-eye image)generated by the signal processor 42 to reconstruct an RGB image(two-dimensional visible image) corresponding to the object, forexample. The depth map generator 44 uses the ML images (compound-eyeimage) resulting from the signal processing by the signal processor 42,the image of the object resulting from the reconstruction by thereconstructing unit 43, or a pixel signal corresponding to reflectedlight from the object, which will be described later, to generate animage (depth map) representing information on the distance in the depthdirection to the object, for example. Specifically, the depth mapgenerator 44 also has functions of a distance information acquiring unitthat acquires the information on the distance in the depth direction tothe object and allows output of the information.

The driver I/F 45 outputs the image (visible image) of the objectresulting from the reconstruction by the reconstructing unit 43 and animage signal representing the depth map generated by the depth mapgenerator 44 to a display driver that is not illustrated. The displaydriver displays the visible image taken by the imaging system 1, thedepth map, and the like.

The controller 50 controls the respective components included in theimaging system 1 according to signals or the like resulting from signalprocessing by the signal processor 42, for example. The determining unit52 receives a signal resulting from signal processing by the signalprocessor 42 via the controller 50, determines whether or not the depthmap generator 44 can generate a depth map (or acquire distanceinformation) on the basis of the received signal, and outputs thedetermination result to the controller 50. When light emitted by theirradiator 12 is in an off state, for example, the determining unit 52determines whether or not the depth map generator 44 can acquiredistance information (whether or not the depth map generator 44 cangenerate a depth map) on the basis of a signal resulting fromphotoelectric conversion performed by the image sensor 20). Thedetermining unit 52 may use the ML images (compound-eye image) resultingfrom the signal processing by the signal processor 42, the image of theobject resulting from the reconstruction by the reconstructing unit 43,or a pixel signal corresponding to reflected light from the object todetermine whether or not the depth map generator 44 can acquire thedistance information (whether or not the depth map generator 44 cangenerate a depth map). The controller 50 also controls the components ofthe imaging system 1 according to the determination result from thedetermining unit 52.

The irradiator 12 emits light to project a pattern onto the object,which will be described later. The irradiator 12 may be capable ofswitching ON/OFF, the wavelength, and the intensity of light to beemitted, the magnification of the pattern to be projected, etc.,according to control by the controller 50.

As described above, the imaging system 1 includes a single sensor (theimage sensor 20), a single lens (the imaging optical system 22),microlenses (included in the microlens array 24), and the irradiator 12,and is capable of taking visible images and depth maps.

Next, the configuration of the imaging module 2 will be described indetail. FIG. 2 is a diagram schematically illustrating a cross sectionof the imaging module 2. As illustrated in FIG. 2, the image sensor 20includes a semiconductor substrate 200 and multiple pixels 202 eachhaving a photodiode and formed at the top of the semiconductor substrate200. The pixels 202 each have a color filter 204 of R, G, B or W, forexample, at the top. The color filters 204 of R have high transmissivityto light in the red wavelength range. The color filters 204 of G havehigh transmissivity to light in the green wavelength range. The colorfilters 204 of B have high transmissivity to light in the bluewavelength range. The color filters 204 of W have light in thewavelength range including the red wavelength range, the greenwavelength range, and the blue wavelength range. The rule forarrangement of the color filter 204 will be described later. Inaddition, the color filters 204 may each have at the top thereof amicrolens 206 for collecting light to each pixel.

A microlens array 24 is arranged above the color filters 204. Themicrolens array 24 includes a visible light transmitting substrate 240and a microlens unit 242 formed thereon, for example. The microlens unit242 is provided on the side of the visible light transmitting substrate240 facing the image sensor 20, and includes multiple microlenses 244arranged two-dimensionally in an array with a predetermined pitch, forexample.

The microlenses 244 each correspond to a pixel block including multiplepixels 202 provided on the semiconductor substrate 200, and function asoptical systems that scale down and focus (collect) light onto thecorresponding pixel blocks. Each pixel block is a block in which 20 to30 pixels are arranged in the diametrical direction or along a sidethereof, for example. Each pixel block may have a structure including 10to 100 pixels arranged in the diametrical direction or along a sidethereof, for example, depending on the use of the image sensor 20.

Furthermore, the visible light transmitting substrate 240 is connectedto the semiconductor substrate 200 by a spacer 270 made of a resinmaterial provided around an imaging region in which the pixels 202 areformed. Note that alignment for connecting the semiconductor substrate200 and the visible light transmitting substrate 240 are carried outwith reference to an alignment mark or the like that are notillustrated.

An optical filter 272 may be provided on the microlens array 24. Forexample, when the visible light transmitting substrate 240 does not havea function of blocking light in an unnecessary wavelength range, anoptical filter having a function of blocking light in an unnecessarywavelength range may be arranged. Light in an unnecessary wavelengthrange refers to light in a wavelength range other than the wavelengthranges transmitted by the color filters 204, for example. In order todistinguish red (R) and near-infrared (NIR) from each other, forexample, an optical filter for blocking light in a wavelength rangethere between may be used.

Furthermore, the semiconductor substrate 200 is provided with anelectrode pad 274 allowing signals output from the pixels 202 to beread. A through electrode 278 extending through the semiconductorsubstrate 200 and making the semiconductor substrate 200 in electriccontinuity with a processing and driving chip 276 is formed under theelectrode pad 274.

The semiconductor substrate 200 is electrically connected to theprocessing and driving chip 276 via the through-electrode 278 and a bump280. The processing and driving chip 276 has formed thereon a drivingand processing circuit (imaging circuit 26) that drives the image sensor20 and processes a signal read from the image sensor 20. Note that theconnection between the semiconductor substrate 200 and the processingand driving chip 276 is not limited to the electric connection via thethrough-electrode 278, but electrode pads provided on the semiconductorsubstrate 200 and the processing and driving chip 276 (the respectivechips) may be connected by a metal wire or the like.

Furthermore, the imaging optical system (imaging lens) 22 is providedabove the visible light transmitting substrate 240. The imaging opticalsystem 22 may include a plurality of imaging lenses. The imaging opticalsystem 22 is attached to a lens barrel 282. The lens barrel 282 isattached to a lens holder 284. The attachment position of the imagingoptical system 22 may be adjusted by the pressing pressure of the lensholder 284 when the lens holder 284 is attached. It is possible todetermine the attachment position of the lens holder 284 while observingan output image on the basis of the relation between the pressingpressure and the output image.

The imaging module 2 also has a light shielding cover 286 attachedthereto to block unnecessary light toward the semiconductor substrate200, the visible light transmitting substrate 240, and the processingand driving chip 276. The imaging module 2 is also provided in a lowerpart of the processing and driving chip 276 with a module electrode 288that electrically connect the processing and driving chip 276 and anexternal device.

Next, geometric optical relations in an optical system (virtual imageoptical system) in the imaging device 10 will be described. FIGS. 3A and3B are diagrams illustrating the geometric optical relations in theoptical system of the imaging device 10. In FIGS. 3A and 3B, forsimplicity, only a range in the vicinity of the optical axes of imaginglenses is drawn. FIG. 3B is an enlarged view of the vicinity of theoptical axes of the microlenses 244 illustrated in FIG. 3A.

In a case where only the imaging optical system (imaging optical system22) is considered, a principal ray 600 and peripheral rays 602 that arerays of the same group as the principal ray 600 from an object point Pon the optical axes are focused by the imaging optical system 22 onto avirtual imaging plane S determined by the focal length f of the imagingoptical system and the distance A between the imaging optical system 22and the object point P so that the relation of the following Expression1 is satisfied:

$\begin{matrix}{\frac{1}{f} = {\frac{1}{A} + \frac{1}{B}}} & (1)\end{matrix}$

In the expression, f represents the focal distance of the imagingoptical system (imaging lens) 22, A represents the distance from aprincipal plane of the imaging optical system 22 facing the object tothe object point P, and B represents the distance from a principal planeof the imaging optical system 22 facing the image to a virtual imagingpoint P′. Note that the principal plane facing the object and theprincipal plane facing the image of the imaging optical system 22 arecoincident in the figures. In addition, an image magnification(horizontal magnification) M of the imaging optical system 22 isexpressed by the following Expression 2:

$\begin{matrix}{M = \frac{B}{A}} & (2)\end{matrix}$

Note that the virtual imaging point P′ of the imaging optical system 22is located behind the image sensor 20 (opposite to the object). In thiscase, since the microlenses 244 are arranged in front of the virtualimaging point P′ (closer to the object than the virtual imaging pointP′), light is collected to the surface on which the pixels 202 of theimage sensor 20 located in front of the virtual imaging plane S areprovided. Thus, the light ray group (the principal ray 600 and theperipheral rays 602) is scaled down and focused in a virtual imagerelationship. The imaging system of the microlenses 244 is expressed bythe following Expression 3:

$\begin{matrix}{\frac{1}{g} = {\frac{1}{C} + \frac{1}{D}}} & (3)\end{matrix}$

In the expression, g represents the focal length of the microlenses 244,C represents the distance from the principal plane of the microlenses244 facing the object to the virtual imaging point P′, and D representsthe distance from the principal plane of the microlenses 244 facing theimage to the imaging point of the microlenses 244. In this case, theimage magnification (image reduction ratio) N of the imaging system ofthe microlenses 244 is expressed by the following Expression 4:

$\begin{matrix}{N = \frac{D}{C}} & (4)\end{matrix}$

The image reduction ratio of the reduction of an image formed by theimaging optical system 22 by the microlens array 24 is not smaller than0.001 and not larger than 0.87, for example. Here, a variable E of thefollowing Expression 5 is introduced on the basis of a geometricalrelation. When the optical system is a fixed focus optical system, thevariable E is a fixed designed value.

E=B−C   (5)

When two adjacent microlenses 244 are selected, the arrangement pitch ofthe microlenses 244 or the distance between the microlenses 244 isrepresented by L_(ML). In this case, the group of light rays 604 a, 604b, 604 c, and 606 coming from the same object are distributed andfocused to multiple imaging points p1, p2, p3, . . . by multiplemicrolenses 244 adjacent to one another. L_(ML) and the image shiftamount Δ on one side are expressed by the following Expression 6 on thebasis of the geometrical relation of the principal rays 604 a, 604 b,and 604 c for the respective microlenses 244 illustrated in FIG. 3B:

$\begin{matrix}{\frac{C}{L_{ML}} = \frac{D}{\Delta}} & (6)\end{matrix}$

Furthermore, the distance A between the object and the imaging opticalsystem 22 and the image shift amount Δ satisfy the relation expressed bythe following Expression 7 according to the expressions (1), (2) and(6):

$\begin{matrix}{A = {\left( {\frac{1}{f} - \frac{1}{B}} \right)^{- 1} = {\left( {\frac{1}{f} - \frac{1}{E + C}} \right)^{- 1} = \left( {\frac{1}{f} - \frac{1}{E + \frac{{DL}_{ML}}{\Delta}}} \right)^{- 1}}}} & (7)\end{matrix}$

Note that f, E, and L_(ML) are designed parameters and are thereforeknown fixed values. Thus, the image shift amount Δ and the distance Dare uniquely determined for the distance A by the above Expression 7.Since the amount of change in the distance D is very small compared tothe amount of change in the distance A, the distance D is assumed to bea fixed value D0. The fixed value D0 represents the distance from theprincipal plane of the microlenses 244 facing the image to the surfaceof the image sensor 20. In this case, the above Expression 7 isexpressed as in the following Expression 8:

$\begin{matrix}{A = {\left( {\frac{1}{f} - \frac{1}{B}} \right)^{- 1} = {\left( {\frac{1}{f} - \frac{1}{E + C}} \right)^{- 1} = \left( {\frac{1}{f} - \frac{1}{E + \frac{D_{0}L_{ML}}{\Delta}}} \right)^{- 1}}}} & (8)\end{matrix}$

In the expression, since f, E, D0, and L_(ML) are known designed values,the distance A to the object can be calculated if the image shift amountΔ can be detected from the surface of the image sensor 20. To obtain theimage shift amount Δ between the image points p1, p2, p3, . . . whenlight rays from the same object point P are focused to the image pointsp1, p2, p3, . . . by the imaging optical system 22 and the microlenses244, image matching between adjacent microlens images (ML images) takenby the image sensor 20 is used.

For the image matching, a known template matching method of finding outsimilarity or dissimilarity between two images can be used, for example.Furthermore, for obtaining shift positions with higher accuracy, thesimilarities obtained for the respective pixels may be interpolated byusing a continuous fitting function or the like, and sub-pixel positionswhere the fitting function is the greatest and the smallest may beobtained to obtain the shift amounts with high accuracy.

Next, a method of reconstructing a two-dimensional visible image by thereconstructing unit 43 will be described. FIG. 4 is a diagramschematically illustrating the method of reconstructing atwo-dimensional visible image by the reconstructing unit 43. Thereconstructing unit 43 reconstructs a two dimensional image with nooverlap from a group of microlens images (ML images) obtained by takingthe same object multiple times.

Assume a case in which three microlenses 244 adjacent to one anotherform microlens images 610 a, 610 b, and 610 c on the surface of theimage sensor 20 as illustrated in FIG. 4, for example. In this manner,to form microlens images that do not overlap with one another, it isonly necessary that the effective F number of the imaging optical system22 and the effective F number of the microlenses 244 be the same.

The fields of view in which the microlens images 610 a, 610 b, and 610 care formed are a field of view 612 a, a field of view 612 b, and a fieldof view 612 c, respectively, on the virtual imaging plane S, which areranges overlapping with one another as illustrated in FIG. 4. FIG. 4illustrates a case in which the image reduction ratio N is 0.5. Thus, asa result of multiplying the respective fields of view by 0.5, everyobject point is taken twice or more times.

When the relation N=0.5 is met, an image on the virtual imaging plane Scan be reproduced by multiplying each microlens image by 1/N, that is,2. To obtain the image reduction ratio N from the group of microlensimages resulting from imaging, derivation of the following Expression 9from the relations of the above expressions (4) and (6) is used:

$\begin{matrix}{N = {\frac{D}{C} = \frac{\Delta}{L_{ML}}}} & (9)\end{matrix}$

Since the pitch L_(LM) of the microlenses 244 is known, the imagereduction ratio N can be obtained when the shift amount Δ of the sameobject is obtained from the images.

When images are combined to reconstruct a two-dimensional image, thereconstructing unit 43 first performs a white balance process ofadjusting the balance of B, G, and R signals on a compound-eye RAW imageoutput by the image sensor 20. Subsequently, since there is no signalinformation of G and B at the position of an R pixel, for example, thereconstructing unit 43 performs a demosaicing process of referring topixels arranged around the R pixel and generating G and B signalsestimated from the pixels. In a simple manner, a process of averagingsurrounding pixels may be performed, but various methods can be used,such as referring to pixels in a wider range where necessary. Thereconstructing unit 43 also performs these processes similarly on Gpixels and B pixels.

Subsequently, the reconstructing unit 43 associates pixel signal valuesSp1, Sp2, Sp3, . . . , Spn taken by the image sensor 20 for image pointsp1, p2, p3, . . . pn corresponding to one object point P as illustratedin FIG. 3A with a signal S′p resulting from combination in n-to-onecorrespondence. The association is carried out by detecting the relationof the image shifts Δ and the overlap relations of the fields of viewfrom the images as described above. The reconstructing unit 43 thenperforms combination by a two-dimensional image combining method asfollows to obtain a two-dimensional image.

The pixel signal values Sp1, Sp2, Sp3, . . . , Spn are used for thecombination to obtain a two-dimensional image. Noise values of therespective pixels are represented by Np1, Np2, Np3, . . . , Npn. First,the reconstructing unit 43 performs an luminance correcting process onthe pixel signal values. Details of the luminance correcting processwill be described in detail later, in which the reconstructing unit 43multiplies the pixels signal values Sp1, Sp2, Sp3, . . . , Spn byluminance correction coefficients a1, a2, a3, . . . , an, respectively,determined by a method to be described later. Subsequently, thereconstructing unit 43 averages the values resulting from themultiplication as expressed by the following Expression 10 to obtain acombined signal value S′p. The noise values contained in the combinedsignal value in this case are similarly multiplied by coefficients, andare thus expressed by the following Expression 11.

S′ _(p) ={a ₁ S _(p1) +a ₂ S _(p2) + . . . +a _(n) S _(pn) }/n   (10)

N′ _(p) ={a ² ₁ N ² _(p1) +a ² ₂ N ² _(p2) + . . . +a ² _(n) N ²_(pn)}^(0.5) /n   (11)

Next, a method of acquiring distance information and generating a depthmap by the depth map generator 44 will be described. As illustrated inFIGS. 3A and 3B, the distance A between the object and the imagingoptical system 22 and the image reduction ratio N are in a one-to-onerelationship determined according to geometric optics. Thus, when ashift amount Δ is obtained by the above Expression 9, the imagereduction ratio N can be obtained since the pitch L_(ML) of themicrolenses 244 is a designed fixed value. Thus, the distance A from theobject can be obtained from the relational expression of the distance Aand the image reduction ratio N.

To obtain the shift amount A of an object from images, an image matchingprocess of searching for and determining the positions of the sameobject points in adjacent microlens images is performed as illustratedin FIG. 5A. In this case, as illustrated in FIG. 5B, the ML image (level1) nearest to the ML image to be searched for is used for calculation ofa low magnification (a distant object) and the second nearest and thethird nearest ML images (levels 2 and 3) are used for calculation of ahigh magnification (a nearby object). Furthermore, as illustrated inFIG. 5B, variation in searching may be reduced by obtaining shiftamounts at six azimuth angles at 60 degree intervals and averaging orobtaining a median of the shift amounts to increase the accuracy to theML image to be searched for, for example. The depth map generator 44then obtains the shift amounts through the image matching processdescribed above.

Next, the irradiator 12 will be described in detail. FIG. 6 is a diagramillustrating an example of relative positions of the imaging device 10and irradiators 12 in the imaging system 1 and the object. In theexample illustrated in FIG. 6, the imaging system 1 is provided with oneirradiator 12 at each of the left and right sides of the imaging device10.

The wavelength range of light that the irradiators 12 emit to project apattern is a visible range, for example. Alternatively, the wavelengthrange of light from the irradiators 12 may be the near-infrared range(wavelength of about 750 nm to 950 nm), which will be described later.In the imaging device 10, the wavelength of light that can be subjectedto photoelectric conversion by the image sensor 20 (that is, the filterproperty) is set depending on the wavelength of light emitted by theirradiators 12.

As illustrated in FIG. 6, the imaging system 1 may be provided with aplurality of irradiators 12 so that no shadow (dead angle) of projectionto the object placed in the field of view of the imaging device 10 willbe produced. Furthermore, the imaging system 1 may be designed toproject a pattern from the side of the imaging module 2 in a manner thatthe angle of view of the projection and the angle of view of imagingmatch with each other, and configured to emit light by one irradiator12.

Next, effects of projecting a pattern onto the object by the irradiator12 will be described. FIGS. 7A and 7D illustrate ML images, FIGS. 7B and7E illustrate reconstructed images, and FIGS. 7C and 7F illustrate depthmaps, respectively, which are different depending on whether or not theirradiator 12 projects a pattern to the object. FIGS. 7A to 7C are a MLimage, a reconstructed image, and a depth map when the irradiator 12projects a pattern onto the object. FIGS. 7D to 7F are a ML image, areconstructed image, and a depth map when the irradiator 12 does notproject a pattern onto the object.

When a pattern is not projected, it is difficult to detect the distanceto the object at a part other than the edge (where the texture changes).The area where it is difficult to detect the distance is the blue areain FIGS. 7C and 7F. In contrast, if a pattern is projected, an objecthaving no texture can also be subjected to the matching process, whichallows the distance to be detected. Specifically, when a pattern isprojected onto the object as illustrated in FIG. 7C, the distance(depth) can be detected all over the object. The light blue area in FIG.7C is the area where distance can be detected. Furthermore, asillustrated in FIG. 7E, the distance is erroneously detected in an areawith much halation, and the distance is not detected in an area with notexture such as the body of the object. The area where the distance iserroneously detected is presented in red in FIG. 7F.

Next, the pattern to be projected by the light emitted by the irradiator12 will be described in detail. While the “pattern” mentioned in thedescription below refers to a white and black stripe pattern in which apair of white and black constitute one period for simplicity in thedescription below, the “pattern” is not limited thereto and may be adifferent pattern.

FIGS. 8A and 8B illustrate part of an ML image (multiple ML images) andpart of a reconstructed image, respectively. The ML image is formed onthe surface of the image sensor 20. The reconstructed image is an imagegenerated by the reconstructing unit 43.

An image of the object that is reduced by a magnification expressed bythe following Expression 12 is formed by the imaging optical system 22on the virtual imaging plane:

M=f/(A−f)   (12)

The image is further reduced at a ratio of N=D/C by the microlens array24. Thus, with the system of the imaging system 1, the object is reducedat a ratio of M×N in total and imaged.

When the pattern image is reduced at a ratio of M×N, that is, by themagnification M of the imaging optical system 22 and by themagnification (reduction ratio) N of the microlenses 244, if thefrequency of the reduced image is not larger than 1/2 of the Nyquistfrequency in the microlenses 244 (not too small) and not smaller than1/2L_(ML) (not too large), image matching can be performed by using thecorresponding ML images and the distance can also be estimated. Thus,when the object is placed in front and the magnification N is large, itis necessary that the pattern will not be lost in gray in the microlensimages. Furthermore, to prevent false detection in the case wherematching is performed, the pattern is preferably a random pattern (anon-periodic structure in the vertical and horizontal directions) or apseudorandom pattern that is close to a random pattern.

FIG. 9 illustrate an ML image with a high black occupancy ratio in thepattern and an ML image with a low black occupancy ratio in the patternin order to explain the suitable size of the projected pattern. Theimaging module 2 is a compound-eye imaging module including themicrolens array 24 in which multiple microlenses 244 are provided.

When the frequency of an image formed on the image sensor 20 is fp(1/mm), the lower limit of the pattern image frequency (the upper limitof the pattern size 1/fp) preferably satisfies the following Expression13 in order to prevent the pattern from being lost because of the sizeof the pattern larger than the ML image:

f _(p)>1/L _(ML)   (13)

In the expression, L_(ML) represents the pitch (mm) of the microlenses244.

Furthermore, the upper limit of the pattern image frequency (the lowerlimit of the pattern size 1/fps) preferably satisfies the followingExpression 14 in order to prevent the pattern from becoming too small toresolve:

f _(p)<1/2d _(pix)   (14)

In the expression, dpix represents the pixel size (mm).

The manner in which the irradiator 12 emits light when a pattern of asize satisfying the upper limit and the lower limit expressed by theabove Expressions 13 and 14 is projected will be described below. Whenthe frequency of the pattern projected onto the object is Fpt (1/mm),distance measurement can be carried out on the corresponding microlensimages if the relation between the pattern Fpt (1/mm) projected onto theobject and the image frequency fp (1/mm) formed on the image sensor 20satisfies the following Expressions 15 and 16.

f _(P) =F _(pt) ×M×N   (15)

1/L _(ML) <F _(pt) ×M×N<1/2d _(pix)   (16)

Thus, the above Expression 16 can also be expressed as in the followingExpression 17.

$\begin{matrix}{L_{ML} > \frac{1}{f_{p}} > {2d_{pix}}} & (17)\end{matrix}$

If all the microlenses 244 that form corresponding images satisfy theaforementioned condition, the condition under which distance can bemeasured is satisfied all over the corresponding image (screen).

FIG. 10A and 10B are diagrams schematically illustrating the shapes ofthe pattern suitable for the imaging system 1. When a projected pattern(texture) is parallel to search axes (epipolar lines) 700 connecting thecenters of the microlens images, the shift amounts cannot be defined.Thus, the pattern is preferably projected in a state tilted between thesearch axes 700. Specifically, if the pattern is anisotropic, theirradiator 12 conducts irradiation so that the direction of the patternis not parallel to the search axes.

Next, specific examples of the pattern will be described. Assume thatthe imaging system 1 performs reduction by a magnification N of about1/2 to 1/10, for example. In this case, the pattern preferably has ashape combining multiple periods so that the texture (pattern) will bedetected, whatever magnification in the range of 1/2 to 1/10 is used forreduction.

FIG. 11 illustrates a pattern combining random patterns with differentperiods. The pattern is suitably a pattern combining a random pattern ofa certain period and random patterns of 1/2 and 1/4 of the period asillustrated in FIG. 11, for example.

Furthermore, as illustrated in FIGS. 12 to 14, such a pattern combiningperiods of squares, triangles, hexagons, or the like is also suitablefor the imaging system 1 since a white and black pattern (a pattern in aframe) of the same periods can be seen, whatever magnification is usedfor reduction. Thus, the irradiator 12 may emit light projecting afractal (self-similar) pattern.

Furthermore, when the irradiator 12 can change the size of the patternto be projected like a projector or the like, the controller 50 maycontrol light emitted by the irradiator 12 so as to change the patternsize on the basis of an image taken by the imaging module 2.

Next, the determining unit 52 will be described. FIG. 15 is a flowchartillustrating an example of operation of the imaging system 1 thatoperates on the basis of the texture of the object. In step 100 (S100),the controller 50 acquires multiple ML images (compound-eye image) of aP-th frame (P is an integer representing the number of frames) via thesignal processor 42.

In step 102 (S102), the controller 50 causes the determining unit 52 todetermine whether or not the depth map generator 44 can acquire distanceinformation (whether or not the depth map generator 44 can generate adepth map) from light from the object for every predetermined Q frames(Q is an integer representing a period using the number of frames).Specifically, the controller 50 determines whether or not the result ofcomputation (P mod Q) is 0. If the result of computation is not 0 (S102:No), the controller 50 causes the ISP 4 (the depth map generator 44) tocarry out processing in S104. If the result of computation is 0 (S102:Yes), the controller 50 proceeds to processing in S110.

In step 104 (S104), the ISP 4 (the depth map generator 44) carries out amatching process by using multiple ML images (compound-eye image) at theP-th frame, for example.

In step 106 (S106), the ISP 4 (the depth map generator 44) outputs adepth map.

In step 108 (S108), the controller 50 increments P (P+1) to count thenumber of frames.

In step 110 (S110), the controller 50 controls the irradiator 12 to turnoff emission of light for projecting a pattern by the irradiator 12.

In step 112 (S112), the determining unit 52 determines whether or nottexture is present on the object. Specifically, the determining unit 52determines whether or not texture is present on the object on the basisof a threshold by using the ML images (compound-eye image) at the P-thframe to determine whether or not the depth map generator 44 can acquirethe distance information (whether or not the depth map generator 44 cangenerate a depth map), for example. If the determining unit 52determines that the depth map generator 44 can acquire the distanceinformation (S112: texture is present), the controller 50 causes the ISP4 (the depth map generator 44) to carry out the processing in S104. Ifthe determining unit 52 determines that the depth map generator 44 cannot acquire the distance information (S112: no (little) texture), thecontroller 50 proceeds to processing in 5114.

In step 114 (S114), the controller 50 controls the irradiator 12 to turnon emission of light for projecting a pattern by the irradiator 12, andproceeds to the processing in S108 (or S100).

Alternatively, the imaging system 1 may first turn off emission of lightfor projecting a pattern by the irradiator 12, acquire a visible image,and carry out texture determination (threshold determination). If thetexture on the object is little (smaller than a predeterminedthreshold), the imaging system 1 then emits light for projecting apattern and outputs a depth map by using reflected light of the pattern.If the texture on the object is sufficient, the imaging system 1 outputsthe depth map without emitting the light for projecting the pattern.

Alternatively, the imaging system 1 may be configured to adjust the gainof the image sensor 20 on the basis of the contrast of the patterndetermined from a read image when the light for projecting the patternis emitted, or may be configured to carry out automatic gain control onthe intensity of light emitted by the irradiator 12. The controller 50may control ON/OFF of the irradiator 12 to cause the irradiator 12 tosimply operate intermittently.

As described above, when the imaging system 1 is provided with thedetermining unit 52, ON/OFF of the irradiator 12 can be determineddepending on the scene of use such as the brightness of the environmentand the colors of the object. The imaging system 1 can therefore reducepower consumption as compared to the case where the irradiator 12continuously emits light (ON) (the case where the irradiator 12 is acompletely active device). Furthermore, with the imaging system 1, thedependency of the texture of the object, the influence of illumination,and the like on the scene is decreased and the distance information canbe acquired more frequently as compared to a completely passive devicewithout any light source.

FIRST MODIFIED EXAMPLE

Next, a first modified example of the imaging system 1 will bedescribed. FIGS. 16A to 16C are diagrams illustrating an outline of thefirst modified example (main part of an imaging module 2 a) of theimaging system 1. FIG. 16A illustrates a partial cross section of theimaging module 2 a. FIG. 16B is an enlarged view around color filters204 a in FIG. 16A. FIG. 16C illustrates an example of an array of thecolor filters 204 a. Note that components in the imaging module 2 aillustrated in FIGS. 16A to 16C which are substantially the same asthose in the imaging module 2 (FIG. 2, etc.) are represented by the samereference numerals.

In the first modified example of the imaging system 1, the irradiator12, which is not illustrated, emits light in the near-infrared (NIR:invisible rays) wavelength range of about 750 to 900 nm to project apattern in the near-infrared range onto the object, and the imagingmodule 2 a reads the pattern in the near-infrared range. An image sensor20 a includes color filters 204 a each provided with a filter of R, G,B, or NIR for each pixel.

When the image sensor 20 a outputs light only from pixels provided withthe NIR filters, the image captured by the image capturing unit 41 is animage of the pattern in the near-infrared range projected onto theobject. The depth map generator 44 then generates a depth map by usingthe image of the pattern in the near-infrared range. The reconstructingunit 43 then demosaics the pixels provided with filters of R, G, and B,and reconstructs a visible image by carrying out interpolation usingperipheral pixels for pixels provided with the NIR filters. In thiscase, only light in the visible range is read but the pattern in thenear-infrared range is not read in the visible image. Thus, according tothe first modified example of the imaging system 1, an image of theobject as viewed by human eyes can be taken.

SECOND MODIFIED EXAMPLE

Next, a second modified example of the imaging system 1 will bedescribed. FIGS. 17A and 17B are diagrams illustrating an outline of thesecond modified example (main part of an imaging module 2 b) of theimaging system 1. FIG. 17A illustrates a partial cross section of theimaging module 2 b. FIG. 17B is an enlarged view around pixels 202 inFIG. 17A. Note that components in the imaging module 2 b illustrated inFIGS. 17A and 17B which are substantially the same as those in theimaging module 2 (FIG. 2, etc.) are represented by the same referencenumerals.

In the second modified example of the imaging system 1, the irradiator12, which is not illustrated, emits light in the near-infrared (NIR)wavelength range of about 750 to 900 nm to project a pattern in thenear-infrared range onto the object, and the imaging module 2 b readsthe pattern in the near-infrared range. A microlens array 24 a includesmicrolens color filters (ML color filters) 246 each provided with afilter of R, G, B or NIR for each microlens 244. According to the secondmodified example of the imaging system 1, a depth map can be generatedby carrying out pattern matching on images formed by the microlenses 244provided with NIR filters, and a visible image can be reconstructed byusing images formed by the microlenses 244 provided with filters of R,G, and B.

As described above, with the imaging system according to the embodiment,since the irradiator is controlled so that images contained in a patternthat is reflected by the object and scaled down on the image sensor bythe imaging optical system and the microlenses are smaller than thearrangement pitch of images each formed on the image sensor by eachmicrolens and larger than twice the pixel, the information on thedistance in the depth direction to the object can be acquired withoutdegrading the accuracy owing to misalignment in installation and withoutdepending on the object.

Thus, the imaging system 1 can also estimate the distance to an objecthaving no texture. Furthermore, since the imaging system 1 includes onecamera, that is, a single sensor (image sensor 20) and has a base linelength that is not between cameras but that is determined by theintervals between adjacent microlenses formed with high accuracyaccording to a micromachining technology, it is not necessary to performalignment between cameras and with a light source and it is possible toprevent degradation in the accuracy caused by misalignment. Furthermore,the imaging system 1 can achieve lower power consumption by making theirradiator 12 operate intermittently or periodically.

Furthermore, since it is only necessary that certain texture beprojected onto the object, interference does not occur even whenmultiple imaging systems 1 are used at the same time. As describedabove, the imaging system 1 can acquire a visible image and a depth mapat the same time by a single device, achieve a smaller size and lowerpower consumption, and improve the probability of distance measurementas a result of being less dependent on the object.

The imaging system 1 can be more easily embedded in various devices as aresult of being smaller in size, which contributes to miniaturization ofproducts such as portable digital assistants and home electricappliances. Furthermore, the imaging system 1 can be applied to thefield of machine vision in which embedded components are required to below in power consumption, small and lightweight such as those mounted invarious industrial robots, robot arms, endoscopes, etc.

While certain embodiments have been described, these embodiments havebeen presented by way of example only, and are not intended to limit thescope of the inventions. Indeed, the novel embodiments described hereinmay be embodied in a variety of other forms; furthermore, variousomissions, substitutions and changes in the form of the embodimentsdescribed herein may be made without departing from the spirit of theinventions. The accompanying claims and their equivalents are intendedto cover such forms or modifications as would fall within the scope andspirit of the inventions.

What is claimed is:
 1. An imaging system comprising: an image sensorthat includes multiple pixel blocks each containing multiple pixelsconfigured to carry out photoelectric conversion; an imaging lensconfigured to focus light from an object onto a virtual imaging plane; amicrolens array that is provided between the image sensor and theimaging lens and includes multiple microlenses arranged with apredetermined pitch, the microlenses being respectively associated withthe pixel blocks; an irradiator configured to emit light to project apattern onto the object; a distance information acquiring unitconfigured to acquire information on distance in a depth direction tothe object on the basis of a signal resulting from photoelectricconversion performed by the image sensor; and a controller configured tocontrol the irradiator so that the pattern formed on the image sensorsatisfies the following expressions (1) and (2):fp=Fpt×M×N   (1), and1/L _(ML) <Fpt×M×N<1/2dpix   (2), where fp represents a frequency of animage formed on the image sensor, Fpt represents a frequency of thepattern, M represents a magnification of the imaging lens, N representsa magnification of the microlenses, L_(ML) represents a distance betweenthe microlenses, and dpix represents a pixel size.
 2. The systemaccording to claim 1, wherein the irradiator emits light to project aperiodic pattern combining a plurality of different periods or a patterncombining a plurality of random patterns of different periods.
 3. Thesystem according to claim 1, wherein the irradiator emits light toproject a fractal pattern.
 4. The system according to claim 1, whereinthe controller performs control to switch on and off of light emitted bythe irradiator.
 5. The system according to claim 4, further comprising adetermining unit configured to determine whether or not the distanceinformation acquiring unit can acquire the information on distance onthe basis of a signal resulting from photoelectric conversion performedby the image sensor when the light emitted by the irradiator is turnedoff, wherein when the determining unit determines that the distanceinformation acquiring unit cannot acquire the information on distance,the controller performs control to switch on the light emitted by theirradiator.
 6. The system according to claim 1, wherein the irradiatoremits invisible light rays to project the pattern, and the image sensorperforms photoelectric conversion on the invisible light rays by some ofthe pixels via filters transmitting the invisible light rays reflectedby the object in a detectable manner.